Processing system and processing method of image data

ABSTRACT

A processor of a first terminal generates composite image data, in which main and sub image data are composited together, based on the main image data, the sub image data, and data about layout of a screen of a second terminal. The main image data are image data which are obtained by a main camera and are to be displayed on a main screen of a display of the second terminal. The sub image data are image data which are obtained by a sub camera and are to be displayed on a sub-screen of the display. A processor of the first terminal transmits the composite image data from the first terminal to the second terminal. A processor of the second terminal displays the composite image data received from the first terminal on the display.

PROCESSING SYSTEM AND PROCESSING METHOD OF IMAGE DATA

The present disclosure claims priority under 35 U.S.C. § 119 to JapanesePatent Application No. 2021-21582, filed on Feb. 15, 2022, the contentsof which application are incorporated herein by reference in theirentirety.

TECHNICAL FIELD

The present disclosure relates to a system and a method of processingimage data transmitted from a first terminal to a second terminal.

BACKGROUND

JP 2014-150299 A discloses a system in which two kinds of image datatransmitted from a first terminal to a second terminal via a network arerespectively displayed on a main screen and a sub-screen of a display ofthe second terminal. In the system in the prior art, positioninformation of the sub-screen in the main screen is transmitted from thesecond terminal to the first terminal. The first terminal performs imageprocessing to fill, with a single color, a specific region in an imagefor the main screen based on the position information. Then, the firstterminal transmits a data set of image data for the sub-screen and imagedata for the main screen, resulting from the image processing, to thesecond terminal. The second terminal performs a picture-in-pictureprocessing (PiP processing) by using the data set and superimposes animage for the sub-screen in a specific region. Image data resulting fromthe PiP processing are displayed on the display of the second terminal.

In the above system in the prior art, a specific region in an image fora main screen is filled in an image processing in a first terminal.Thus, compared to a case where two kinds of image data for the mainscreen and the sub-screen are transmitted without any change, a dataamount to be transmitted can be reduced. However, because two kinds ofimage data are transmitted, when the data amount of one kind of imagedata becomes large, transmission at a designated frame rate becomesdifficult. Consequently, when image data for the main screen and thesub-screen are displayed on a display of the second terminal, it isdesired to make an improvement for further reduction in the data amountto be transmitted from the first terminal to the second terminal.

One object of the present disclosure is to provide a technique that canreduce a data amount to be transmitted from a first terminal to a secondterminal when image data for a main screen and a sub-screen aredisplayed on a display of the second terminal.

SUMMARY

A first aspect of the present disclosure provides an image dataprocessing system causing image data transmitted from a first terminalto a second terminal via a network to be displayed on a display of thesecond terminal, the image data processing system including thefollowing features.

The first terminal includes a data processing device and a communicationdevice which communicates with the second terminal. The data processingdevice of the first terminal includes a memory, which stores main imagedata which are obtained by a main camera and are to be displayed on amain screen of the display and sub image data which are obtained by asub camera and are to be displayed on a sub-screen of the display, and aprocessor.

The second terminal includes a data processing device and acommunication device which communicates with the first terminal. Thedata processing device of the second terminal includes a memory, whichstores data about layout of a screen of the second terminal, and aprocessor.

The processor of the first terminal is configured to:

execute a data synthesizing processing to generate composite image data,in which the main and sub image data are composited together, based onthe main and sub image data and the data about the layout which arereceived from the second terminal; and

a data transmission processing to transmit the composite image data tothe second terminal.

The processor of the second terminal is configured to execute displaycontrol processing to display on the display the composite image datareceived from the first terminal.

A second aspect of the present disclosure further includes the followingfeatures in the first aspect.

The processor of the first terminal is further configured to executeobject recognition processing to recognize an object in a periphery ofthe first terminal based on the main image data.

In the data synthesizing processing, the processor of the first terminalis further configured to:

determine whether or not a dynamic object recognized in the objectrecognition processing is positioned in a predetermined region in a mainimage to which the sub image data are planned to be joined; and

execute data adjustment processing to visualize the dynamic object whenit is determined that the dynamic object is positioned in thepredetermined region.

A third aspect of the present disclosure further includes the followingfeatures in the second aspect.

The data adjustment processing includes a stop of joining of the subimage data, change in a position of the predetermined region, andshrinking of a size of the predetermined region.

A fourth aspect of the present disclosure further includes the followingfeatures in the first aspect.

The first terminal further includes a data base that stores additionalimage data which are set while being associated with a moving situationof the first terminal.

The memory of the first terminal further stores data about the movingsituation.

In the data synthesizing processing, the processor of the first terminalis further configured to:

extract the additional image data corresponding to the moving situationby referring to the data base based on the data about the movingsituation; and

superimpose the additional image data corresponding to the movingsituation on the main image data.

A fifth aspect of the present disclosure further includes the followingfeatures in the fourth aspect.

In the data synthesizing processing, the processor of the first terminalis configured to superimpose the additional image data on a region otherthan a predetermined region in a main image to which the sub image dataare planned to be joined.

A sixth aspect of the present disclosure further includes the followingfeatures in the first aspect.

The first terminal is a terminal of a moving body on which the main andsub cameras are mounted.

The second terminal is a terminal for performing remote assistance forthe moving body.

A seventh aspect of the present disclosure provides an image dataprocessing method of causing image data transmitted from a firstterminal to a second terminal via a network to be displayed on a displayof the second terminal.

The image data processing method comprising the steps of:

the first terminal generating composite image data, in which main andsub image data are composited together, based on the main image datawhich are obtained by a main camera and are to be displayed on a mainscreen of the display, the sub image data which are obtained by a subcamera and are to be displayed on a sub-screen of the display, and dataabout layout of a screen of the second terminal;

the first terminal transmitting the composite image data to the secondterminal; and

the second terminal displaying the composite image data received fromthe first terminal on the display.

An eighth aspect of the present disclosure further includes thefollowing features in the seventh aspect.

The image data processing method further comprising the steps of:

the first terminal recognizing an object in a periphery of the firstterminal based on the main image data;

the first terminal determining whether or not a dynamic objectrecognized in the step of recognizing an object in the periphery of thefirst terminal is positioned in a predetermined region in a main imageto which the sub image data are planned to be joined; and

the first terminal executing data adjustment processing to notsynthesize the sub image data with the predetermined region when it isdetermined that the dynamic object is positioned in the predeterminedregion.

A ninth aspect of the present disclosure further includes the followingfeatures in the eighth aspect.

The data adjustment processing includes a stop of joining of the subimage data, change in a position of the predetermined region, andshrinking of a size of the predetermined region.

A tenth aspect of the present disclosure further includes the followingfeatures in the seventh aspect.

The first terminal is a terminal of a moving body on which the main andsub cameras are mounted.

The second terminal is a terminal for performing remote assistance forthe moving body.

According to the first or seventh aspect, the first terminal executesthe processing (data synthesizing processing) to synthesize the mainimage data with the sub image data and the processing (data transmissionprocessing) to transmit the composite image data to the second terminal.Consequently, compared to a case where the main image data and the subimage data are transmitted to the second terminal without any change, itbecomes possible to reduce a data amount of image data to be transmittedfrom the first terminal to the second terminal.

According to the second or eighth aspect, when a dynamic object isrecognized in the object recognition processing performed based on themain image data, it is determined whether or not the dynamic object ispositioned in the predetermined region in the main image to which thesub image data are planned to be joined. Then, when it is determinedthat the dynamic object is positioned in the predetermined region, thedata adjustment processing is executed to visualize the dynamic object.Consequently, when the dynamic object is positioned in the predeterminedregion in the main image, it becomes possible to avoid trouble that thedynamic object does not appear on the composite image.

According to the third or ninth aspect, by a stop of joining of the subimage data to the main image data, change in the position of thepredetermined region, and shrinking of the size of the predeterminedregion, it becomes possible to avoid trouble that the dynamic objectdoes not appear on the composite image.

According to the fourth aspect, in the data synthesizing processing, theadditional image data which are set while being associated with themoving situation of the first terminal can be superimposed on the mainimage data. Consequently, when augmented reality information isdisplayed on the display, it becomes possible to reduce the data amountto be transmitted from the first terminal to the second terminal.

According to the fifth aspect, in the data synthesizing processing, theadditional image data can be superimposed on a region other than thepredetermined region in the main image to which the sub image data areplanned to be joined. Consequently, when an additional image issuperimposed on the main image, it becomes possible to avoid troublethat a sub image does not appear due to the additional image.

According to the sixth or tenth aspect, when the remote assistance forthe moving body is executed by the first and second terminal, it becomespossible to reduce the data amount of image data to be transmitted fromthe first terminal to the second terminal.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram for explaining an outline of a processing example ofimage data which is executed in a first embodiment of the presentdisclosure;

FIG. 2 is a diagram for explaining a processing example in prior artwhich is compared to the processing example of the first embodiment;

FIG. 3 is a block diagram illustrating a configuration example of afirst terminal in the first embodiment;

FIG. 4 is a diagram for explaining examples of a main camera and a subcamera;

FIG. 5 is a block diagram illustrating a configuration example of asecond terminal in the first embodiment;

FIG. 6 is a flowchart illustrating a flow of a data processing exampleto be executed by a data processing device of the first terminal in thefirst embodiment;

FIG. 7 is a diagram for explaining one example of a data synthesizingprocessing which is executed in the first embodiment;

FIG. 8 is a flowchart illustrating a flow of a data processing exampleto be executed by a data processing device of the second terminal in thefirst embodiment;

FIG. 9 is a diagram for explaining an outline of a processing example ofimage data which is executed in a second embodiment of the presentdisclosure;

FIG. 10 is a flowchart illustrating a flow of a data processing exampleto be executed by the data processing device of the first terminal inthe second embodiment;

FIG. 11 is a diagram for explaining an outline of a processing exampleof image data which is executed in a third embodiment of the presentdisclosure;

FIG. 12 is a block diagram illustrating a configuration example of thefirst terminal in the third embodiment;

FIG. 13 is a diagram for explaining one example of a data synthesizingprocessing which is executed in the third embodiment;

FIG. 14 is a diagram illustrating a display example of composite imagedata to be displayed on a display when a display control processing isexecuted in the second terminal; and

FIG. 15 is a diagram illustrating another display example of thecomposite image data to be displayed on the display when the displaycontrol processing is executed in the second terminal.

DESCRIPTION OF EMBODIMENT

Processing systems and processing methods of image data according toembodiments of the present disclosure will hereinafter be described withreference to drawings. Note that the processing methods according to theembodiments are realized by computer processing to be executed in theprocessing systems according to the embodiments. Further, the samereference characters are given to the same or corresponding componentsin the drawings, and descriptions thereof will be simplified or will notbe repeated.

1. First Embodiment

A first embodiment of the present disclosure will be described withreference to FIGS. 1 to 8 .

1-1. Outline of First Embodiment

FIG. 1 is a diagram for explaining an outline of a processing example ofimage data which is executed in the first embodiment. FIG. 2 is adiagram for explaining a processing example in prior art which iscompared to the processing example of the first embodiment.

FIG. 1 illustrates a first terminal 1 and a second terminal 2. The firstterminal 1 communicates with the second terminal 2 via a network (notillustrated). In the example illustrated in FIG. 1 , as data to betransmitted from the first terminal 1 to the second terminal 2,composite image data IMG_C are illustrated. The composite image dataIMG_C are image data that can be obtained by compositing together mainimage data IMG_M and sub image data IMG_S which are obtained at the sametime. The main image data IMG_M are image data obtained by a maincamera. The sub image data IMG_S are image data obtained by a subcamera. The main camera and the sub camera may be mounted on the firstterminal or may be mounted on an external device which is capable ofcommunicating with the first terminal.

The composite image data IMG_C transmitted from the first terminal 1 tothe second terminal 2 are displayed on a display 21 of the secondterminal 2. The display 21 is divided into a main screen 21M and asub-screen 21S. In the example illustrated in FIG. 1 , the sub-screen21S is a quadrangular region positioned in an upper center of thedisplay 21. In a display region of the sub-screen 21S, image datacorresponding to the sub image data IMG_S of the composite image dataIMG_C are displayed. Meanwhile, in a display region of the main screen21M, image data are displayed in which the image data corresponding tothe display region of the sub-screen 21S are removed from the main imagedata IMG_M of the composite image data IMG_C.

Similarly to FIG. 1 , FIG. 2 illustrates a first terminal 3 whichcommunicates with a second terminal 4 via a network. In a processingexample of the prior art which is illustrated in FIG. 2 , as data to betransmitted from the first terminal 3 to the second terminal 4, the mainimage data IMG_M and the sub image data IMG_S are illustrated. In otherwords, in the processing example of the prior art, the main image dataIMG_M and the sub image data IMG_S are transmitted to the secondterminal 4 without any change.

In the processing example of the prior art, the second terminal 4 whichreceives the main image data IMG_M and the sub image data IMG_Sgenerates the composite image data IMG_C. Generation of the compositeimage data IMG_C is executed by a PiP processing, for example. Thecomposite image data IMG_C generated by the PiP processing are displayedon a display 41 of the second terminal 4. In a display region of asub-screen 41S of the display 41, image data corresponding to the subimage data IMG_S are displayed. Meanwhile, in a display region of a mainscreen 41M, image data are displayed in which the image datacorresponding to the display region of the sub-screen 41S are removedfrom the main image data IMG_M.

A large difference between the processing example of the firstembodiment and that in the prior art is the terminal in which thecomposite image data IMG_C are generated. That is, in the former, theterminal on a transmission side (in other words, the first terminal 1)generates the composite image data IMG_C, but in the latter, theterminal on a reception side (in other words, the second terminal 4)generates the composite image data IMG_C. In the former, such adifference makes it possible to reduce the data amount to be transmittedfrom the first terminal 1 to the second terminal 2.

In the following, a description will be made about a processing systemaccording to the first embodiment.

1-2. Processing System 1-2-1. Configuration Example

A description will be made about a configuration example of theprocessing system according to the first embodiment with reference toFIGS. 3 to 5 . FIG. 3 is a block diagram illustrating a configurationexample of the first terminal 1 in the first embodiment. As illustratedin FIG. 3 , the first terminal 1 includes a main camera 11, a sub camera12, a communication device 13, and a data processing device 14.Configuration elements such as the main camera 11 are connected with thedata processing device 14 by a predetermined network.

The main camera 11 is a camera for obtaining the main image data IMG_M.The sub camera 12 is a camera for obtaining the sub image data IMG_S.The main camera 11 transmits the obtained main image data IMG_M to thedata processing device 14. The sub camera 12 transmits the obtained subimage data IMG_S to the data processing device 14.

The numbers of main cameras 11 and sub cameras 12 are not particularlylimited, and one camera may be provided, or two or more cameras may beprovided. FIG. 4 is a diagram for explaining examples of the main camera11 and the sub camera 12. In the example illustrated in FIG. 4 , maincameras 11A to 11C and sub cameras 12A to 12C are mounted on a movingbody MV. The moving body MV is one example of an external device whichis capable of communicating with the first terminal 1.

Regions indicated by broken lines in FIG. 4 correspond to respectiveranges in which those cameras capture images. In the example illustratedin FIG. 4 , the main camera 11A captures an image of a front center ofthe moving body MV. The main camera 11B captures an image of front rightof the moving body MV. The main camera 11C captures an image of frontleft of the moving body MV. The sub camera 12A captures an image of rearof the moving body MV. The sub camera 12B captures images of a rightside and right rear of the moving body MV. The sub camera 12C capturesimages of a left side and left rear of the moving body MV.

For example, one main camera 11 is combined with one sub camera 12. Inthe example illustrated in FIG. 4 , the main camera 11A is combined withthe sub camera 12A. Further, the main camera 11B is combined with thesub camera 12B, and the main camera 11C is combined with the sub camera12C.

Returning to FIG. 3 , the description about the configuration example ofthe processing system will be continued. The communication device 13executes wireless communication with a base station (not illustrated) ona network. As a communication standard of this wireless communication, astandard of mobile communication such as 4G, LTE, or 5G may be raised asan example. Connection destinations of the communication device 13include at least the second terminal 2. In communication with the secondterminal 2, the communication device 13 transmits at least the compositeimage data IMG_C to the second terminal 2. The communication device 13may transmit ID data and present position data of the first terminal 1(or the moving body MV) to the second terminal 2.

The data processing device 14 is a computer which executes dataprocessing based on various kinds of data provided to the first terminal1 and various kinds of data obtained by the first terminal 1. The dataprocessing device 14 includes at least one processor 14 a and at leastone memory 14 b. The processor 14 a includes a central processing unit(CPU). The memory 14 b is a volatile memory such as a DDR memory,expands various programs to be used by the processor 14 a, andtemporarily saves various kinds of data. The various kinds of datainclude the main image data IMG_M, the sub image data IMG_S, and thecomposite image data IMG_C.

The processor 14 a executes a predetermined program for data processing,which is stored in the memory 14 b, and thereby executes various kindsof data processing. The various kinds of data processing include “datasynthesizing processing” for generating the composite image data IMG_Cand “data transmission processing” for transmitting the composite imagedata IMG_C to the second terminal 2 via the communication device 13.Details of the data synthesizing processing and the data transmissionprocessing will be described later.

FIG. 5 is a block diagram illustrating a configuration example of thesecond terminal 2 in the first embodiment. As illustrated in FIG. 5 ,the second terminal 2 includes the display 21, an input device 22, acommunication device 23, and a data processing device 24. Configurationelements such as the display 21 are connected with the data processingdevice 24 by a predetermined network.

The display 21 displays the composite image data IMG_C received from thefirst terminal 1. When the first terminal 1 is a terminal mounted on themoving body MV illustrated in FIG. 4 , the second terminal 2 is aterminal for executing remote assistance for the moving body MV, forexample, and the composite image data IMG_C displayed on the display 21are monitored by an operator who executes the remote assistance.

Here, as the remote assistance by the operator, recognition assistanceand decision assistance may be raised as example. For example, when themoving body MV executes autonomous driving, when sunlight shines on atraffic light present in front of the moving body MV, precision ofrecognition of a lighting state of light emitting units of the trafficlight is lowered. When the lighting state cannot be recognized, itbecomes difficult to decide what kind of behavior has to be executed atwhich timing. In such a case, recognition assistance about the lightingstate and/or decision assistance about behavior of the moving body MVare executed, the decision assistance being based on the lighting staterecognized by the operator.

The remote assistance by the operator also includes remote driving. Inthe remote driving, the operator recognizes the composite image dataIMG_C displayed on the display 21 and executes a driving operation ofthe moving body MV which includes at least one of steering,acceleration, and deceleration.

The input device 22 is operated by a user of the second terminal 2. Theinput device 22 includes an input unit which is operated by the user anda control circuit which generates and outputs instruction data INS basedon input data, for example. As input units, a mouse, a keyboard, abutton, and a switch may be raised as examples. As the instruction dataINS, data for changing (such as enlarging and shrinking) display formsof the composite image data IMG_C on the display 21, data for changingsettings of the display regions of the main screen 21M and thesub-screen 21S, and so forth may be raised as examples.

When the second terminal 2 executes the remote assistance for the movingbody MV, the input device 22 is operated by the operator. In this case,the instruction data INS may be transmitted to the first terminal 1.When the operator executes remote driving of the moving body MV, theinput device 22 may include input devices for traveling. As the inputdevices for traveling, a steering wheel, a shift lever, an acceleratorpedal, and a brake pedal may be raised as examples. When the inputdevices for traveling are operated by the operator, the instruction dataINS are transmitted to the first terminal 1.

The communication device 23 executes wireless communication with a basestation on a network. As a communication standard of this wirelesscommunication, a standard of mobile communication such as 4G, LTE, or 5Gmay be raised as an example. Connection destinations of thecommunication device 23 include at least the first terminal 1. Incommunication with the first terminal 1, the communication device 23transmits data (hereinafter, also referred to as “layout data”) LAYabout layout of the display 21 to the first terminal 1. As the layoutdata LAY, data about sizes of the display 21 in horizontal and verticaldirections, data about sizes of the display region of the sub-screen 21Sin the horizontal and vertical directions, and data of the position ofthe display region may be raised as examples. When the second terminal 2executes the remote assistance for the moving body MV, the communicationdevice 23 may transmit the instruction data INS to the first terminal 1.

The data processing device 24 is a computer which executes dataprocessing based on various kinds of data provided to the secondterminal 2 and various kinds of data obtained by the second terminal 2.The data processing device 24 includes at least one processor 24 a andat least one memory 24 b. The processor 24 a includes a CPU. The memory24 b is a volatile memory such as a DDR memory, expands various programsto be used by the processor 24 a, and temporarily saves various kinds ofdata. The various kinds of data include the layout data LAY and theinstruction data INS.

The processor 24 a executes a predetermined program for data processing,which is stored in the memory 24 b, and thereby executes various kindsof data processing. The various kinds of data processing include“display control processing” for displaying the composite image dataIMG_C received from the first terminal 1 on the display 21. Details ofthe display control processing will be described later.

1-2-2. Data Processing Example

FIG. 6 is a flowchart illustrating a flow of a data processing examplewhich is executed in the data processing device 14 (processor 14 a). Aroutine illustrated in FIG. 6 is repeatedly executed in a predeterminedcycle (for example, a reception interval of the main image data IMG_M).

In the routine illustrated in FIG. 6 , various kinds of data are firstobtained (step S11). The various kinds of data include the main imagedata IMG_M and the sub image data IMG_S which are obtained at the sametime. The various kinds of data also include the layout data LAYreceived from the second terminal 2.

Following the processing in step S11, the data synthesizing processingis executed (step S12). The data synthesizing processing is executedbased on the main image data IMG_M, the sub image data IMG_S, and thelayout data LAY which are obtained in step S11.

FIG. 7 is a diagram for explaining one example of the data synthesizingprocessing. In the example illustrated in FIG. 7 , based on the layoutdata LAY, size and position of a display region AR_S, to which the subimage data IMG_S are planned to be joined, are first specified. In theexample illustrated in FIG. 7 , the display region AR_S is positioned inan upper center of the main image data IMG_M.

In the example illustrated in FIG. 7 , next, image data in the mainimage which correspond to the position of the display region AR_S aredeleted. This deletion processing contributes to reduction in a dataamount of the main image data IMG_M. Further, in parallel with thisdeletion processing, adjustment of the sub image data IMG_S obtained atthe same time as the main image data IMG_M is executed. In the exampleillustrated in FIG. 7 , the size of the sub image data IMG_S is shrunkso as to conform to the size of the display region AR_S. This shrinkingprocess contributes to reduction in a data amount of the sub image dataIMG_S. After the shrinking process, image data of an outer peripheryportion of the sub image data IMG_S may be deleted so as to conform tothe size of the display region AR_S.

In the example illustrated in FIG. 7 , next, the adjusted sub image dataIMG_S are joined (inserted) to a position in the upper center of themain image data IMG_M (in other words, the position of the displayregion AR_S). Accordingly, the composite image data IMG_C are generated.

Returning to FIG. 6 , following the processing in step S12, the datatransmission processing is executed (step S13). In the data transmissionprocessing, encoding processing of the composite image data IMG_C isexecuted, and the composite image data IMG_C are output to thecommunication device 13. In the encoding processing, the composite imagedata IMG_C may be compressed. The composite image data IMG_C output tothe communication device 13 are transmitted from the first terminal 1 tothe second terminal 2.

FIG. 8 is a flowchart illustrating a flow of a data processing examplewhich is executed in the data processing device 24 (processor 24 a). Aroutine illustrated in FIG. 8 is repeatedly executed in a predeterminedcycle (for example, a reception interval of the composite image dataIMG_C).

In the routine illustrated in FIG. 8 , various kinds of data are firstobtained (step S21). The various kinds of data include the compositeimage data IMG_C received from the first terminal 1.

Following the processing in step S21, the display control processing isexecuted (step S22). In the display control processing, decodingprocessing of the composite image data IMG_C obtained in step S21 isexecuted, and the composite image data IMG_C are output to the display21. When the composite image data IMG_C are compressed, the data aredecompressed in the decoding process.

1-3. Effects

In the above-described first embodiment, the processing (datasynthesizing processing) to synthesize the main image data IMG_M withthe sub image data IMG_S is executed in the first terminal 1, andthereafter the transmission processing (data transmission processing) totransmit the composite image data IMG_C to the second terminal 2 isexecuted. Consequently, compared to a case where the main image dataIMG_M and the sub image data IMG_S are transmitted to the secondterminal 2 without any change, it becomes possible to reduce the dataamount of image data to be transmitted from the first terminal 1 to thesecond terminal 2.

In particular, when the remote assistance for the moving body MV isexecuted, images of a periphery of the moving body MV have to becaptured by plural cameras, and the data amount of the image data to betransmitted from the first terminal 1 to the second terminal 2 tends tobecome large. When the data amount of the image data to be transmittedbecomes large, transmission at a designated frame rate becomesdifficult, and there is a concern that trouble occurs such as anincrease in communication delay and an occurrence of communicationdisruption. In this point, in the first embodiment, because it becomespossible to inhibit an occurrence of such trouble, contribution tosmooth practice of the remote assistance is expected.

2. Second Embodiment

A second embodiment of the present disclosure will be described withreference to FIGS. 9 and 10 . Note that in the following, descriptionsabout contents common to the first embodiment will appropriately beskipped.

2-1. Outline of Second Embodiment

FIG. 9 is a diagram for explaining an outline of a processing example ofimage data which is executed in the second embodiment. Note that in anupper stage of FIG. 9 , the processing example of the first embodimentwhich is compared to the processing example of the second embodiment isillustrated.

In the processing example of the second embodiment, the composite imagedata IMG_C are generated by compositing together the main image dataIMG_M and the sub image data IMG_S which are obtained at the same time.Further, in the processing example in the second embodiment, the subimage data IMG_S whose size and so forth are adjusted are joined(inserted) to the position of the display region AR_S in the main image.To this point, the processing example is the same as the processingexample of the first embodiment.

The processing example of the first embodiment is on the presumptionthat the position of the display region AR_S is fixed. Thus, there is aproblem that when an object is present in the position of the displayregion AR_S, image data of a part or the whole of the object disappearfrom the composite image data IMG_C. As illustrated in the upper stageof FIG. 9 , when a dynamic object MO (for example, a moving body such asa pedestrian, a bicycle, a motorcycle, or a vehicle) is present in theposition of the display region AR_S, a large part of an image of thedynamic object MO does not appear on the composite image.

Such a situation is not desirable for the user of the second terminal 2who checks and observes presence of the dynamic object MO in a peripheryof the first terminal 1 by using the composite image data IMG_C. Whenthe remote assistance for the moving body MV is executed by using thecomposite image data IMG_C, in view of securing traveling safety of themoving body MV, falling into such a situation has to be avoided.

Accordingly, in the processing example of the second embodiment, beforethe composite image data IMG_C are generated, an object which appears inthe main image is recognized by using the main image data IMG_M. Then,when the dynamic object MO is recognized in the position of the displayregion AR_S, data processing (data adjustment processing) forvisualizing the dynamic object MO is executed.

In a lower stage of FIG. 9 , an example of the data adjustmentprocessing is illustrated. In a first example, joining of the sub imagedata IMG_S to the position of the display region AR_S is stopped. Inthis case, the composite image data IMG_C are configured only with themain image data IMG_M.

In a second example, the position of the display region AR_S is changed.In the second example, the display region AR_S which has already beenchanged is positioned on a left side of the display region AR_S whichhas not yet been changed. However, the changed position of the displayregion AR_S is not particularly limited as long as the position is aposition in which the dynamic object MO is not present, and the displayregion AR_S can be moved in an arbitrary direction of up, down, left, orright. The second example may be combined with the data adjustment (stopof joining) described in the first example.

In a third example, the size of the display region AR_S is shrunk. Forexample, a reference point (for example, the center of gravity of aquadrangular region or a left lower or right lower apex) in the displayregion AR_S is fixed, and the size of the display region AR_S isrepeatedly shrunk until the dynamic object MO comes out from theposition of the shrunk display region AR_S. The third example may becombined with the data adjustment (stop of joining) described in thefirst example. The third example may also be combined with the dataadjustment (change in position) described in the second example.

When the data adjustment processing is executed, it becomes possible toavoid trouble that a part or the whole of the image of the dynamicobject MO does not appear on the composite image. This leads toenhancement of visibility of the dynamic object MO by using thecomposite image data IMG_C and also to enhancement of traveling safetyof the moving body MV by using the composite image data IMG_C.

In the following, a description will be made about a processing systemaccording to the second embodiment. Note that a configuration example ofthe processing system is common to the configuration example describedin the first embodiment. Thus, in the following, an example of dataprocessing by the processing system will be described.

2-2. Data Processing Example

FIG. 10 is a flowchart illustrating a flow of a data processing examplewhich is executed in the data processing device 14 (processor 14 a). Aroutine illustrated in FIG. 10 is executed as a subroutine of step S12illustrated in FIG. 6 , for example.

In the routine illustrated in FIG. 10 , an object recognition processingis first executed (step S121). In the object recognition processing,based on the main image data IMG_M, an object in a periphery of the maincamera 11, whose image is captured by the main camera 11, is recognized.As an object to be recognized, in addition to the dynamic object MO, astatic object may be raised as an example. As static objects, aconstruction, a building, and so forth which are present in theperiphery of the main camera 11 may be raised as examples. When the maincamera 11 is mounted on the moving body MV, as the static objects, aline demarcating a lane on which the moving body MV is presentlytraveling, a guardrail, a traffic signal, and so forth may be raised asexamples.

Following the processing in step S121, it is determined whether or notthe dynamic object MO is present in the position of the display regionAR_S (step S122). The position of the display region AR_S is specifiedbased on the layout data LAY received from the second terminal 2. Thus,when the dynamic object MO is recognized in the processing in step S121,a positional relationship between the position of the image of thedynamic object MO in the main image and the position of the displayregion AR_S is calculated. For example, several representative positionsof the image of the dynamic object MO are specified. Then, it isdetermined whether or not all of the representative positions are on theinside of the display region AR_S. When it is determined that all of therepresentative positions are on the inside of the display region AR_S,it is determined that the dynamic object MO is present in the positionof the display region AR_S.

When an affirmative determination result is obtained in step S122, thedata adjustment processing is executed (step S123). As alreadydescribed, the data adjustment processing is data processing forvisualizing the dynamic object MO present in the position of the displayregion AR_S. In the first example of the data adjustment processing,joining of the sub image data IMG_S to the position of the displayregion AR_S is stopped.

In the second example of the data adjustment processing, the position ofthe display region AR_S is changed. In this case, it is determinedwhether or not the dynamic object MO (or another dynamic object MO) ispresent in the position of the display region AR_S whose position hasalready been changed, the display region AR_S is moved to a position inwhich the dynamic object MO (or another dynamic object MO) is notpresent. When even when movement of the display region AR_S is repeated,the dynamic object MO (or another dynamic object MO) is present in theposition of the display region AR_S at a movement destination, joiningof the sub image data IMG_S may be stopped.

In the third example of the data adjustment processing, the size of thedisplay region AR_S is shrunk. In this case, it is determined whether ornot the dynamic object MO (or another dynamic object MO) is present inthe position of the display region AR_S whose size has already beenchanged, the size of the display region AR_S is shrunk until the dynamicobject MO (or another dynamic object MO) comes out from the position ofthe display region AR_S whose size has already been changed. When thesize of the display region AR_S becomes less than a threshold value as aresult of repetition of shrinking of the display region AR_S, joining ofthe sub image data IMG_S may be stopped.

In a fourth example of the data adjustment processing, change in theposition of the display region AR_S and shrinking of the size areexecuted in combination. When even when change in the position and thesize is repeated, the dynamic object MO (or another dynamic object MO)is present in the position of the display region AR_S which has alreadybeen changed, joining itself of the sub image data IMG_S may be stopped.Also when the size of the display region AR_S whose position and sizehave already been changed becomes less than a threshold value, joiningof the sub image data IMG_S may be stopped.

When a negative determination result is obtained in step S122 or afterthe processing in step S123, the composite image data IMG_C aregenerated (step S124). When a negative determination result is obtainedin step S122, the composite image data IMG_C are generated in accordancewith the processing example described in FIG. 7 . When the position ofthe display region AR_S is changed, the sub image data IMG_S whose sizehas already been adjusted are joined (inserted) to the position of thedisplay region AR_S whose position has already been changed. When thesize of the display region AR_S is shrunk, the size of the sub imagedata IMG_S is shrunk so as to conform to the size of the display regionAR_S whose size has already been changed. Subsequently, the sub imagedata IMG_S whose size has already been adjusted are joined (inserted) tothe position of the display region AR_S whose size has already beenchanged. When joining of the sub image data IMG_S is stopped, thecomposite image data IMG_C are generated only from the main image dataIMG_M.

2-3. Effects

In the above-described second embodiment, the data adjustment processingis executed. The data adjustment processing makes it possible to avoidtrouble that a part or the whole of the image of the dynamic object MOdoes not appear on the composite image. This leads to enhancement ofvisibility of the dynamic object MO by using the composite image dataIMG_C and also to enhancement of traveling safety of the moving body MVby using the composite image data IMG_C.

3. Third Embodiment

A third embodiment of the present disclosure will be described withreference to FIGS. 11 to 15 . Note that in the following, descriptionsabout contents common to the first or second embodiment willappropriately be skipped.

3-1. Outline of Third Embodiment

FIG. 11 is a diagram for explaining an outline of a processing exampleof image data which is executed in the third embodiment. FIG. 11illustrates the composite image data IMG_C. The composite image dataIMG_C are data to be transmitted from the first terminal 1 to the secondterminal 2. The composite image data IMG_C include image data that canbe obtained by compositing together the main image data IMG_M and thesub image data IMG_S which are obtained at the same time.

The composite image data IMG_C transmitted from the first terminal 1 tothe second terminal 2 are displayed on the display 21 of the secondterminal 2. In the display region of the sub-screen 21S, image datacorresponding to the sub image data IMG_S of the composite image dataIMG_C are displayed. To this point, the processing example is the sameas the processing example of the first embodiment.

Differently from the processing example of the first embodiment, in theprocessing example of the third embodiment, the composite image dataIMG_C are composited from the main image data IMG_M, the sub image dataIMG_S, and additional image data IMG_A which are obtained at the sametime. The additional image data IMG_A are image data which are set whilebeing associated with a moving situation of the first terminal 1. Theadditional image data IMG_A are generated based on the data about themoving situation of the first terminal 1.

Further, in the processing example of the third embodiment, the display21 includes a display region of an augmented reality information screen21A in addition to the display regions of the main screen 21M and thesub-screen 21S. In other words, the display region of the main screen21M of the third embodiment is decreased by the amount of the displayregion of the augmented reality information screen 21A compared to thedisplay region of the main screen 21M in the first embodiment. In thedisplay region of the augmented reality information screen 21A, imagedata corresponding to the additional image data IMG_A of the compositeimage data IMG_C are displayed. In the display region of the main screen21M, image data are displayed in which the image data corresponding tothe display regions of the sub-screen 21S and the augmented realityinformation screen 21A are removed from the main image data IMG_M of thecomposite image data IMG_C.

In such a manner, in the processing example of the third embodiment, thecomposite image data IMG_C including the additional image data IMG_A aregenerated in the first terminal 1 and transmitted to the second terminal2. Consequently, when the augmented reality information is displayed onthe display 21, it becomes possible to reduce the data amount to betransmitted from the first terminal 1 to the second terminal 2.

In the following, a description will be made about a processing systemaccording to the third embodiment.

3-2. Processing System 3-2-1. Configuration Example

FIG. 12 is a block diagram illustrating a configuration example of thefirst terminal 1 in the third embodiment. As illustrated in FIG. 12 ,the first terminal 1 includes the main camera 11, the sub camera 12, thecommunication device 13, the data processing device 14, a sensor group15, and a data base 16. In other words, the configuration exampleillustrated in FIG. 12 is a configuration example in which the sensorgroup 15 and the data base 16 are added to the configuration example inthe first embodiment which is described in FIG. 3 . Configurationelements such as the main camera 11 are connected with the dataprocessing device 14 by a predetermined network.

The sensor group 15 includes sensors which detect the moving situationof the first terminal 1. As the sensors which detect the movingsituation, a speed sensor and an acceleration sensor may be raised asexamples. The speed sensor detects a speed of the first terminal 1. Theacceleration sensor detects an acceleration of the first terminal 1. Thesensors which detect the moving situation also include a positionsensor. The position sensor detects the position and a bearing of thefirst terminal 1. As the position sensor, a global navigation satellitesystem (GNSS) sensor may be raised as an example. Various kinds ofmoving situation data STS detected by those sensors are transmitted tothe data processing device 14.

When the first terminal 1 is a terminal mounted on the moving body MVillustrated in FIG. 4 , the sensors which detect the moving situationmay include a yaw rate sensor and a steering angle sensor in addition tothe speed sensor and the acceleration sensor. In this case, the yaw ratesensor detects a yaw rate around a perpendicular axis of the center ofgravity of the moving body MV. The steering angle sensor detects anangle of a steering wheel of the moving body MV. When the first terminal1 is a terminal mounted on the moving body MV illustrated in FIG. 4 ,the sensor group 15 may also include recognition sensors other than themain camera 11 and the sub camera 12. A recognition sensor recognizes anambient environment of the moving body MV by using an electric wave orlight. As the recognition sensors, a millimeter-wave radar and laserimaging, detection, and ranging (LIDAR) may be raised as examples.

The data base 16 stores various kinds of data necessary for generationof the additional image data IMG_A. The various kinds of data, speedimage data, acceleration image data, and map image data may be raised asexamples. Generation of the additional image data IMG_A by using thespeed image data is executed by referring to the moving situation dataSTS (for example, the speed of the first terminal 1). Generation of theadditional image data IMG_A by using the acceleration image data isexecuted by referring to the moving situation data STS (for example, theacceleration of the first terminal 1). Generation of the additionalimage data IMG_A by using the map image data is executed by referring tothe moving situation data STS (for example, the position of the firstterminal 1).

When the first terminal 1 is a terminal mounted on the moving body MVillustrated in FIG. 4 , various kinds of data to be stored in the database 16 include image data which indicate a future traveling track ofthe moving body MV. Generation of the additional image data IMG_A byusing track image data is executed by predicting the future travelingtrack of the moving body MV based on the moving situation data STS (forexample, the speed and steering angle of the moving body MV) and byextracting the track image data closest to the shape of the travelingtrack from the data base 16.

A configuration example of the second terminal 2 in the third embodimentis the same as the configuration example in the first embodiment whichis described in FIG. 5 .

3-2-2. Data Processing Example

A processing example of image data in the third embodiment is basicallythe same as the processing example in the first embodiment which isdescribed in FIGS. 6 to 8 . FIG. 13 is a diagram for explaining oneexample of a data synthesizing processing which is executed in the thirdembodiment. The data synthesizing processing illustrated in FIG. 13 isexecuted in step S12 in FIG. 6 , for example.

In the example illustrated in FIG. 13 , based on the layout data LAY,size and position of a display region AR_S, to which the sub image dataIMG_S are planned to be joined, are first specified. Further, image datain the main image which correspond to the position of the display regionAR_S are deleted. Further, in parallel with this deletion processing,adjustment of the sub image data IMG_S obtained at the same time as themain image data IMG_M is executed. Then, the adjusted sub image dataIMG_S are joined (inserted) to the position of the specified displayregion AR_S. To this point, the processing example is the same as theprocessing example described in FIG. 7 .

In the example illustrated in FIG. 13 , next, a display region AR_A towhich the additional image data IMG_A are planned to be joined isspecified. The display region AR_A is specified in a region, from whichthe display region AR_S to which the sub image data IMG_S are joined isremoved. Then, the additional image data IMG_A are superimposed on theposition of the specified display region AR_A.

3-3. Display Examples of Composite Image Data

FIGS. 14 and 15 are diagrams illustrating display examples of thecomposite image data IMG_C to be displayed on the display 21 when thedisplay control processing is executed in the second terminal 2. Notethat FIGS. 14 and 15 illustrate examples of the composite image dataIMG_C when the second terminal 2 is a terminal for executing the remoteassistance for the moving body MV. Further, FIGS. 14 and 15 do notillustrate display of the sub-screen 21S.

In the example illustrated in FIG. 14 , in the position of the displayregion AR_A, the composite image data IMG_C to which the additionalimage data IMG_A are joined are displayed, the additional image dataIMG_A corresponding to the moving situation data STS of the moving bodyMV such as a speed and an engine rotation speed. Meanwhile, in theexample illustrated in FIG. 15 , the composite image data IMG_C aredisplayed in which the additional image data IMG_A for guiding parkingof the moving body MV are superimposed on the main image data IMG_M.

3-4. Effects

In the above-described third embodiment, the composite image data IMG_Cincluding the additional image data IMG_A are generated in the firstterminal 1 and transmitted to the second terminal 2. Consequently, whenthe augmented reality information is displayed on the display 21, itbecomes possible to reduce the data amount to be transmitted from thefirst terminal 1 to the second terminal 2.

What is claimed is:
 1. An image data processing system causing imagedata transmitted from a first terminal to a second terminal via anetwork to be displayed on a display of the second terminal, wherein thefirst terminal includes a data processing device and a communicationdevice which communicates with the second terminal, wherein the dataprocessing device of the first terminal includes a processor and amemory which stores main image data which are obtained by a main cameraand are to be displayed on a main screen of the display and sub imagedata which are obtained by a sub camera and are to be displayed on asub-screen of the display, wherein the second terminal includes a dataprocessing device and a communication device which communicates with thefirst terminal, wherein the data processing device of the secondterminal includes a processor and a memory which stores data aboutlayout of a screen of the second terminal, wherein: the processor of thefirst terminal is configured to: execute a data synthesizing processingto generate composite image data, in which the main and sub image dataare composited together, based on the main and sub image data and thedata about the layout which are received from the second terminal; and adata transmission processing to transmit the composite image data to thesecond terminal, the processor of the second terminal is configured toexecute display control processing to display on the display thecomposite image data received from the first terminal.
 2. The image dataprocessing system according to claim 1, wherein the processor of thefirst terminal is further configured to execute object recognitionprocessing to recognize an object in a periphery of the first terminalbased on the main image data, wherein, in the data synthesizingprocessing, the processor of the first terminal is further configuredto: determine whether or not a dynamic object recognized in the objectrecognition processing is positioned in a predetermined region in a mainimage to which the sub image data are planned to be joined; and executedata adjustment processing to visualize the dynamic object when it isdetermined that the dynamic object is positioned in the predeterminedregion.
 3. The image data processing system according to claim 2,wherein the data adjustment processing includes a stop of joining of thesub image data, change in a position of the predetermined region, andshrinking of a size of the predetermined region.
 4. The image dataprocessing system according to claim 1, wherein the first terminalfurther includes a data base that stores additional image data which areset while being associated with a moving situation of the firstterminal, wherein the memory of the first terminal further stores dataabout the moving situation, wherein, in the data synthesizingprocessing, the processor of the first terminal is further configuredto: extract the additional image data corresponding to the movingsituation by referring to the data base based on the data about themoving situation; and superimpose the additional image datacorresponding to the moving situation on the main image data.
 5. Theimage data processing system according to claim 4, wherein, in the datasynthesizing processing, the processor of the first terminal isconfigured to superimpose the additional image data on a region otherthan a predetermined region in a main image to which the sub image dataare planned to be joined.
 6. The image data processing system accordingto claim 1, wherein: the first terminal is a terminal of a moving bodyon which the main and sub cameras are mounted; and the second terminalis a terminal for performing remote assistance for the moving body. 7.An image data processing method of causing image data transmitted from afirst terminal to a second terminal via a network to be displayed on adisplay of the second terminal, the method comprising the steps of: thefirst terminal generating composite image data, in which main and subimage data are composited together, based on the main image data whichare obtained by a main camera and are to be displayed on a main screenof the display, the sub image data which are obtained by a sub cameraand are to be displayed on a sub-screen of the display, and data aboutlayout of a screen of the second terminal; the first terminaltransmitting the composite image data to the second terminal; and thesecond terminal displaying the composite image data received from thefirst terminal on the display.
 8. The method according to claim 7,further comprising the steps of: the first terminal recognizing anobject in a periphery of the first terminal based on the main imagedata; the first terminal determining whether or not a dynamic objectrecognized in the step of recognizing an object in the periphery of thefirst terminal is positioned in a predetermined region in a main imageto which the sub image data are planned to be joined; and the firstterminal executing data adjustment processing to not synthesize the subimage data with the predetermined region when it is determined that thedynamic object is positioned in the predetermined region.
 9. The methodaccording to claim 8, wherein the data adjustment processing includes astop of joining of the sub image data, change in a position of thepredetermined region, and shrinking of a size of the predeterminedregion.
 10. The method according to claim 7, wherein: the first terminalis a terminal of a moving body on which the main and sub cameras aremounted; and the second terminal is a terminal for performing remoteassistance for the moving body.